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EXECUTIVE  SUMMARY 


“i insights  into  tomorrow ” 


Part  of  our  College  mission  is  distribution  of  the 
students’  problem  solving  products  to  DoD 
sponsors  and  other  interested  agencies  to 
enhance  insight  into  contemporary,  defense 
related  issues.  While  the  College  has  accepted  this 
product  as  meeting  academic  requirements  for 
graduation,  the  views  and  opinions  expressed  or 
implied  are  solely  those  of  the  author  and  should 
not  be  construed  as  carrying  official  sanction. 


REPORT  NUMBER  88-2150 

AUTHOR(S) MAJOR  STEPHEN  C.  POPP,  USAF 

TITLE  ADEQUACY  OF  GUIDANCE  TO  CONDUCT  INITIAL  OPERATIONAL 
SUITABILITY  TEST  AND  EVALUATION  OF  INTERCONTINENTAL 
BALLISTIC  MISSILES  (ICBMs) 


I.  Purpose :  To  determine  i  -f  adequate  guidance  exists  to  develop  test 
objectives,  supporting  tests,  and  evaluation  methodologies  -for  initial 
operational  suitability  test  and  evaluation  of  land-based  ICBMs. 

II.  Probl em ;  Operational  test  and  evaluation  is  not  a  new  concept  for 
the  Air  Force;  however,  the  majority  of  the  test  programs  and  guidance 
written  to  conduct  test  programs  is  oriented  to  aircraft  and  aircraft 
systems.  While  the  Peacekeeper  in  Minuteman  Silos  program  is  drawing  to 
a  close,  two  new  programs  are  in  the  planning  phase:  Peacekeeper  Rail 
Garrison  and  the  Small  ICBM.  Operational  suitability  objectives, 
supporting  tests,  and  evaluation  methodologies  are  required  to  support 
these  programs.  Therefore,  a  determination  of  the  adequacy  of  guidance 
provided  to  develop  test  objectives,  supporting  tests,  and  evaluation 
methodologies  for  initial  operational  suitability  test  and  evaluation  of 
land-based  ICBMs  must  be  made. 

III.  Data :  Guidance  and  direction  to  conduct  operational  suitability 
test  and  evaluation  comes  from  Department  of  Defense  and  Air  Force 
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regulations.  Direction  to  develop  test  objectives,  supporting  tests, 
and  evaluation  methodologies  is  described  in  AFR  55-43,  Management  of 
Operational  Test  and  Evaluation,  with  more  detailed  guidance  being 
provided  in  AFOTEC  regulations  and  pamphlets.  The  process  to  develop 
test  objectives  is  an  analytical  process  where  test  objectives  -flow  -from 
the  critical  operational  issues.  Tests  to  support  the  suitability  test 
objectives  are  developed  -from  measures  o-f  ef-fect  i  veness  <M0E)  which  are 
quantitative  measures  o-f  the  objectives.  Evaluation  methodologies  are 
the  means  to  interpret  test  results.  The  examples  and  procedures  given 
in  the  guidance  are  -from  aircraft  or  aircra-ft  munitions  test  and 
evaluation  programs.  AFOTEC  Pamphlet  400-1,  Looistics  Assessment 
provides  speci-fic  guidance  in  appendices  for  various  systems  undergoing 
evaluation,  but  does  not  have  an  appendix  for  land-based  ICBMs.  The 
Peacekeeper  in  Minuteman  Silos  test  program  developed  classic 
suitability  test  objectives  of  availability,  logistics  reliability, 
maintainability,  and  logistics  suppor tabi 1 i ty,  but  failed  to  develop 
suitability  MOEs  to  support  the  objectives.  The  majority  of  the  tests 
developed  supported  the  maintainability  objective.  Some  tests,  which 
actually  supported  an  objective,  were  not  shown  as  supporting  all 
related  objectives.  Evaluation  methodologies  used  in  the  PIMS  program 
were  those  suggested  by  the  guidance  with  two  notable  exceptions: 
model i ng  and  simul at i on ,  and  reliability  and  maintainability  growth 
techn iques. 


IV.  Conclusions:  The  guidance  to  conduct  operational  suitability  test 
and  evaluation  of  land-based  ICBMs  has  some  shortcomings.  The  failure 
to  address  development  of  MOEs  for  ICBMs  as  separate  and  distinct  from 
aircraft  and  aircraft  systems  may  have  contributed  to  the  absence  of 
suitability  MOEs  in  the  PIMS  test  program.  There  is  also  a  lack  of 
emphasis  in  the  current  guidance  on  development  of  tests  to  support 
suitability  objectives.  The  guidance  emphasizes  test  development  for 
mission  effectiveness  tests,  but  does  not  give  the  same  emphasis  for 
suitability.  Detailed  descriptions  of  evaluation  methodologies  for 
various  systems  are  provided  in  AFOTEC  Pamphlet  400-1,  in  separate 
appendices,  but  no  appendix  exists  for  land-based  ICBMs.  This  may  have 
contributed  to  the  PIMS  program  not  using  some  of  the  suggested 
methodologies.  However,  the  PIMS  program  test  planners  failed  to 
utilize  the  guidance  available. 


V.  Recommendat ions:  The  guidance  to  develop  MOEs  should  recognize  the 
differences  between  ICBMs  and  aircraft  and  include  procedures  to  develop 
MOEs  using  terms  relevant  to  ICBMs.  An  equal  emphasis  on  suitability 
tests  to  support  suitability  test  objectives  is  needed  in  the  guidance. 
The  current  guidance  to  use  simulation  and  modeling  requires  to  much 
subjective  analysis  and  not  enough  on  actual  system  performance.  An 
appendix  should  be  added  to  AFOTEC  Pamphlet  400-1  for  land-based  ICBMs 
addressing  evaluation  methodologies  unique  to  these  weapon  systems.  A 
new  Air  Force  Pamphlet,  AFP  57-9,  Defining  Logistics  Requirement;  in 
Statements  of  Operational  Need  provides  an  example  for  other  guidance 
to  follow.  This  regulation  provides  guidance  separate  from  aircraft  and 
munitions,  with  terms  relevant  to  ICBMs. 


INTRODUCTION 


BACKGROUND 


The  Peacekeeper  <MX)  Intercontinental  Ballistic  Missile  (ICBM)  is 
the  -first  new  ICBM  in  over  twenty  years  and  the  -first  to  be  formally 
tested  by  the  Air  Force  Operational  Test  and  Evaluation  Center(AFOTEC) . 
In  April  of  1983  a  final  decision  was  made  to  deploy  the  Peacekeeper  in 
existing  Minuteman  silos  at  Francis  E.  Warren  AFB,  Wyoming  (3:1-2). 
Following  this  decision,  plans  for  testing  the  system  in  this  deployment 
mode  were  developed.  It  was  also  decided  the  test  program  would  be 
conducted  jointly,  with  initial  operational  testing  integrated  into  the 
developmental  test  and  evaluation  (DT&E)  program.  AFOTEC  had 
responsibility  for  conducting  the  initial  operational  test  and 
eval uat i on< IOT&E)  <3: — ). 

"The  purpose  of  the  Peacekeeper  OT&E  program  is  to  estimate  the 
operational  effectiveness  and  su i tabi 1 i ty.  of  the  weapon  system  in  its 
intended  operational  environment*  (3:1-3).  The  task  for  Headquarters 
AFOTEC  and  the  AFOTEC  test  team  was  to  develop  test  objectives, 
supporting  tests  and  the  methodologies  to  measure  the  performance  of  the 
weapon  system  in  its  intended  operational  environment.  Since  no 
previous  land  based  ICBM  IOT&E  program  had  been  conducted  there  was  no 
precedent  vo  follow.  Department  of  Defense,  Air  Force,  and  AFOTEC 
regulations  provided  the  guidance  to  develop  objectives,  tests,  and 
methodologies.  Additionally,  the  supporting  tests  had  to  be  integrated 
into  the  existing  DT&E  program.  The  objectives  and  tests  also  had  to 
consider  the  intended  environment}  a  new  airframe  in  an  existing 
Minuteman  silo.  Other  considerations  for  IOT&E  test  objectives  and 
tests  are  the  destructive  nature  of  ICBM  testing,  the  extensive  silo 
support  equipment,  continuous  run  requirement,  and  differences  between 
test  facilities  and  operational  sites.  From  the  information  available, 
the  AFOTEC  test  team  developed  the  Peacekeeper  Operational  Test  and 
Evaluation  Plan  describing  the  test  objectives,  supporting  tests,  and 
methodologies  for  the  test  program. 

It  is  the  opinion  of  this  author  that  despite  the  existence  of 
regulations  and  guidance  to  conduct  OT&E,  there  does  not  exist  adequate 
guidance  addressing  land-based  ICBMs.  The  existing  guidance  is  oriented 
towards  aircraft  and  aircraft  systems  or  munitions  used  by  aircraft. 
While  this  lack  of  guidance  affects  both  operational  effectiveness  and 


suitability,  this  paper  will  examine  the  operational  suitability  issue 
only. 


PROBLEM  STATEMENT 

Operational  test  and  evaluation  is  not  a  new  concept  -for  the  Air 
Force.  However,  the  majority  of  the  test  programs  and  guidance  to 
conduct  test  programs  is  oriented  to  aircraft  and  aircraft  systems.  The 
Peacekeeper  test  team  has  completed  17  of  the  scheduled  20  -flight  tests 
and  submitted  an  interim  test  report.  Test  objectives,  tests,  and 
methodologies  were  developed  to  support  the  test  program.  As  the 
Peacekeeper  program  draws  to  a  close  it  is  not  the  end  o-f  initial 
operational  test  and  evaluation  o-f  1CBM  systems.  Two  additional 
programs  are  in  the  test  planning  phase;  Peacekeeper  Rail  Garrison  and 
Small  ICBM.  Operational  suitability  test  objectives,  supporting  tests 
and  methodologies  are  required  to  support  these  programs.  There-fore,  a 
determination  o-f  the  adequacy  o-f  guidance  provided  to  develop  test 
objectives,  supporting  tests,  and  evaluation  methodologies  -for  initial 
operational  suitability  test  and  evaluation  o-f  land  based  ICBMs  must  be 
made.  Should  de-f i c i enc i es  be  noted  recommendations  -for  improved 
guidance  must  be  identified.  The  research  objectives  for  this  paper  are 
presented  in  Table  1-1. 


Operational  test  and  evaluation  is  a  very  broad  concept  covering 
the  entire  life  of  a  system.  It  is  concerned  with  the  operational 
mission  effectiveness  and  operational  suitability  of  a  system.  This 
paper  will  confine  itself  to  only  initial  operational  test  and 
evaluation  and  further  limited  to  operational  suitability  test  and 
evaluation.  The  paper  will  confine  itself  to  only  the  Peacekeeper  in 
Minuteman  Silos  (PIMS)  test  program  and  will  not  consider  previous 
Peacekeeper  basing  modes. 


Objective  1:  To  investigate  Department  of  De-fense  and  Air  Force 

guidance  -for  initial  operational  suitability  test  and 
evaluation. 

Objective  2:  To  investigate  suitability  test  objectives  and 

supporting  tests  used  in  the  Peacekeeper  in  Minuteman 
Silos  <PIMS)  test  program. 

Objective  3:  To  investigate  methodologies  used  in  the  initial 

operational  suitability  test  and  evaluation  of  the  PIMS 
test  program. 

Objective  4:  To  compare  Air  Force  and  DOD  guidance  to  the  PIMS  test 
program. 

Objective  5:  To  develop  recommendations  for  additional  guidance  on 
operational  suitability  test  objectives,  supporting 
tests,  and  evaluation  methodologies  for  future  ICBM 
test  programs. 


Chapter  Two 


CURRENT  GUIDANCE  ON  SUITABILITY 
TEST  AND  EVALUATION 

The  purpose  of  this  section  is  to  present  the  existing  guidance  on 
conducting  operational  suitability  test  and  evaluation.  First,  the 
requirement  for  conducting  operational  suitability  test  and  evaluation 
is  presented,  followed  by  the  current  guidance  on  how  to  develop 
suitability  test  objectives.  The  current  guidance  on  how  to  develop 
actual  tests  to  support  the  test  objectives  is  explored,  concluding  with 
guidance  on  how  to  develop  methodologies  to  evaluate  test  results.  The 
findings  in  this  section  provide  the  basis  to  compare  the  findings  in 
Objectives  Two  and  Three,  draw  conclusions  in  Objective  Four,  and  make 
recommendations  in  Objective  Five  as  outlined  in  Chapter  One. 

REQUIREMENT  FOR  OT&F 

Operational  Test  and  Evaluation  is  directed  by  Department  of 
Defense  Directive  5000.3,  Test  and  Evaluation  and  Air  Force  Regulation 
80-14,  Test  and  Evaluation.  These  Directives  state: 

Their  [test  and  evaluation!  primary  purposes  are:  to 

identify,  assess,  and  reduce  the  acquisition  risks;  to 
evaluate  operational  effectiveness  and  operational 
suitability;  to  id-ntify  any  deficiencies  in  the  system;  and 
to  ensure  that  only  operationally  effective  and  suitable, 
supportable  system  are  delivered  to  the  operating  forces 
<9:2). 

A  further  objective  of  Air  Force  operational  test  and  evaluation  is  to 
'Evaluate  .  .  .  the  operational  suitability  of  the  system'  <9:7),  The 
goal  being  to  measure  '.  .  .  the  system  against  the  operational  criteria 
outlined  in  program  documentation  developed  by  DOD,  HQ  USAF,  and 
operating  and  supporting  commands  (for  example,  system  operational 

concepts)'  (9:3).  From  these  operational  criteria  test  objectives  are 
developed. 
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The  development  of  OT&E  objectives  is  best  viewed  as  an  analytical 
process.  The  process  is  presented  graphically  in  Figure  1.  This 
-figure  is  a  composition  o-f  those  -found  in  Air  Force  Regulation  55-43  and 
AFOTEC  Pamphlet  400-1  .  OT&E  objectives  -flow  -from  the  original  documents 
requesting  or  documenting  a  need  -for  a  new  weapon  system. 

The  original  source  document  usually  is  the  "Statement  o-f 
Operational  Need"  or  SON.  It  is  in  this  document  that  an  Air  Force 
agency  identi-fies  what  it  needs  to  ful-fill  current  mission  requirements 
or  must  have  to  ful-fill  a  new  mission  in  the  future.  The  SON  should, 

".  .  .  express  capabilities  which  the  new  system  is  expected  to  achieve" 
(5:11).  It  is  from  these  requirements  and  capabilities  that  critical 
issues  are  developed. 

"Critical  issues  are  those  aspects  addressing  a  system's  essential 
capabilities,  risks,  or  uncertainties  that  must  be  explained  before  the 
systems  overall  worth  can  be  estimated*  (7:14).  Simply,  critical  issues 
address  the  question,  "Can  we  do  what  we  need  to  do  with  the  proposed 
system?"  For  the  purpose  of  OT&E,  critical  issues  addressing 
operational  issues  are  identified  and  separately  labeled  as  "critical 
operational  issues"  and  used  to  develop  OT&E  objectives. 


STATEMENT  OF  OPERATIONAL  NEED 
(SON) 


CRITICAL  ISSUES 

(CRITICAL  OPERATIONAL  ISSUES) 

\ 

OBJECTIVES 

(SUITABILITY  OBJECTIVES) 

\ 

SUBOBJECTIVES 

\ 

MEASURES  OF  EFFECTIVENESS 

\ 

SUPPORTING  TESTS 


Figure  1.  Flow  Diagram  of  Test  and  Evaluation  Process  (5:13; 
7:18). 
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OT&E  objectives  form  the  -foundation  -for  operational  test  and 
evaluation.  "Objectives  address  the  scope  of  OT&E,  the  critical  issues, 
and  ultimately  determine  the  resources  required*  <7:19).  Hence,  the 
objectives  flow  from  the  critical  operational  issues.  Test  objectives 
usually  address  characteristics  of  the  system  and  are  related  to  the 
critical  issues  <7:19;  5:14).  Subobjectives  may  be  needed  to  further 
breakdown  the  objective  into  quantitative,  measurable  units  addressing 
specific  system  characteristics.  <7:19;  5:14)  The  sub-objectives  form 
the  building  blocks  that  support  an  objective,  which  in  turn,  supports  a 
critical  issue. 

General  guidance  on  the  development  of  objectives  and  sub¬ 
objectives  is  provided  in  AFR  55-43,  Management  of  QT&E.  A  checklist  is 
provided  for  the  planner  to  compare  objectives  and  subobjectives.  More 
specific  guidance  on  actual  development  of  operational  suitability  test 
objectives  is  found  in  AFQTECP  400-1,  Logistics  Assessment.  This 
document  stresses  the  identification  and  separation  of  critical  issues 
relating  to  suitability  and  the  development  of  OT&E  objectives  and  sub¬ 
objectives.  <5:39) 

Measures  of  effectiveness  <MOEs)  form  the  link  between  test 
objectives  and  tests.  "An  MOE  is  a  quantitative  or  qualitative  measure 
of  a  system's  performance  or  characteristic  which  indicates  the  degree 
to  which  it  performs  a  task  or  meets  an  objective  under  specified 
conditions"  <7:22).  An  MOE  is  related  to  a  specific  test  objective.  As 
example,  ah  MOE  may  specify  the  mean  time  between  maintenance  <MTBM) 
<5:14).  The  MOEs  flow  from  the  list  of  functions  the  system  is  to 
perform.  These  MOEs  are  then  tied  to  specific  test  events  <7:22). 
AFOTECP  400-1  <5:14)  states,  "Test  planning  depends  on  developing  MOEs 
that  are  directly  applicable  to  the  test  objectives." 


DEVELOPMENT  OF  SUPPORTING  TESTS 

Major  General  Patrick  Powers  of  the  Army  Test  and  Evaluation 
Command  gave  a  perspective  on  the  importance  of  tests  when  he  stated, 
"One  good  test  is  worth  a  thousand  expert  opinions  and  usually  costs 
less"  <14:35).  Tests  are  developed  to  support  the  QT&E  objectives,  and 
result  from  an  analysis  of  the  missions  the  weapon  system  is  going  to  be 
required  to  perform.  AFR  55-43  provides  guidance  for  the  development  of 
supporting  tests. 

The  greatest  emphasis  in  the  current  test  guidance  is  that  they  are 
as  realistic  and  representative  of  the  actual  operating  environment  as 
possible.  <9:3;  7:14;  5:30)  But,  the  guidance  also  recognizes  the 
limitations  in  conducting  realistic  tests.  <7:14;  5:30)  A  quote  from 
AFR  55-43  typifies  this.  "To  fulfill  the  need  for  operational  realism, 
a  test  scenario  should  approximate  the  intended  operations  environment 
within  the  constraints  imposed  by  safety  considerations,  fiscal 
resources,  the  natural  environment,  and  time  alloted  to  0T&E’  <7:14). 


The  guidance  on  test  development  in  AFR  55-43  relates  mainly  to 
operational  effectiveness  tests.  The  process  described  is  general  in 
nature.  All  missions  the  system  is  to  perform  are  listed  and  -further 
broken  down  into  various  mission  elements  and  the  specific  actions  and 
tasks  in  each  mission  element.  These  tasks  are  then  prioritized  and 
considered  against  the  objectives  and  program  constraints.  (7:77-78) 

The  corresponding  section  under  suitability  test  approach  in  AFR 
55-43  provides  no  guidance  on  how  to  develop  supporting  suitability 
tests  nor  does  it  state  the  necessity  for  supporting  tests.  It  suggests 
using  simulations  and  modeling  because  of  limitations  and  undeveloped 
logistic  support  system  (7:81). 

In  the  literature  concerning  OT&E  tests,  the  same  emphasis  on 
realism  can  be  found  <2:224-225;  14:22;  17:48;  12:91;  15:4).  However, 
there  is  not  the  emphasis  on  testing  limitations.  General  Powers 
believes  testing  will  save  time  and  money  in  the  future  and  calls  for 
more  testing,  not  less.  He  also  thinks  testing  should  not  suffer  due  to 
scheduling  or  time  constraints  (16:158). 

The  literature  does  point  out  some  additional  considerations  for 
suitability  test  planners.  Rear  Admiral  Kollmorgen,  Assistant  Director, 
Strategic  and  Support  Systems  Test  and  Evaluation,  Office  of  Research 
and  Engineering,  points  out  the  dependence  on  laboratory  and  bench 
testing  for  reliability  evaluation  which  fails  to  expose  equipment  and 
systems  to  operational  stress  and  operational  personnel.  He  also  states 
the  need  to  expose  hardware  to  the  cumulative  effects  of  the  operational 
environment  (13:9).  Major  General  Leaf,  former  Commander  of  the  Air 
Force  Test  and  Evaluation  Center,  also  stated  the  need  for  testing  to  be 
*.  .  .  conducted  by  the  operations  and  maintenance  personnel  who  will 
use  the  system  in  the  field.*  (14:21)  Karns  also  states  the  need  for 
testing  utilizing  operational  personnel  (12:91).  General  Leaf  also 
recognized  the  need  for  separate  OT&E  tests  and  that  eventually  DT&E 
tests  must  yield  to  operational  tests  (14:22-23). 

As  testified  above,  experts  agree  that  tests  to  support  operational 
suitability  should  be  as  realistic  as  possible,  utilize  operational 
personnel,  and  be  conducted  in  the  operational  environment.  The  test 
should  be  dedicated  to  OT&E,  separate  from  DT&E  tests,  and  within  the 
limitations  of  the  program  and  other  constraints. 


DEVELOPMENT  OF  EVALUATION  METHODOLOGIES 


Once  supporting  tests  have  been  conducted  and  the  results  o-f  the 
tests  collected,  these  results  must  be  evaluated.  Lieutenant 
Colonel  Robbins  <17:51)  stated  in  a  1974  article: 

Tests  alone  do  not  provide  simple  answers  totally 
applicable  to  operational  reality.  Evaluation  is  needed  to 
apply  reasoning  and  judgement  to  the  test  results  and  answer 
the  operational  questions  about  a  weapon  system's 
effectiveness  and  suitability  (17:51). 

Specific  guidance  is  provided  in  AFOTECP  400-1,  Looi st i cs 
Assessment .  on  the  suitability  evaluation  methodologies.  The  pamphlet 
addresses  common  techniques  to  evaluate  suitability  objectives.  A  brief 
synopsis  of  each  follows. 

Avai lability 

The  assessment  of  availability  determines  if  the  system  is  operable 
and  committable  when  needed  at  a  random  point  in  time.  Availability  is 
a  function  of  reliability,  maintainability,  and  logistics 
supportabi 1 i ty .  It  is  usually  expressed  as  the  percentage  of  assets 
committable  when  called  upon  to  perform  the  required  mission. 
Availability  can  be  mathematically  expressed  as: 

Hours  commi ttable 

Availability  =  Total  hours  possessed 


The  general  approach  to  evaluate  availability  is  to  collect  data  on  how 
often  the  system  breaks  (reliability),  how  long  it  takes  to  fix 
(maintainability),  and  how  responsive  support  equipment,  technical  data, 
facilities,  and  supply  support  are  when  a  demand  is  placed  on  them 
(logistics  supportabi 1 i ty) . 

The  guidance  on  availability  points  out  that  assessment  of 
availability  during  IOT&E  may  be  difficult  due  to  lack  of  operational 
equipment  and  tests  in  an  operational  environment.  The  recommended 
actions  to  assist  in  the  evaluation  of  availability  is  to  use  modeling 
or  simulation  <5:Ch  16,  A7-1  -  A7-2) . 

Reliability 

The  assessment  of  reliability  determines  the  frequency  a  system 
will  fail  when  operated.  Quantitatively,  it  can  be  expressed  as  a 
probability  the  system  will  work  for  the  time  required  to  perform  its 
intended  mission.  It  can  also  be  expressed  as  mean  time  between 
maintenance  (MTBM) ,  mean  time  between  removal  (MTBR) ,  or  mean  time 
between  critical  failure  (MTBCF).  Other  measures  can  be  used,  but  these 
are  the  most  common.  Data  requirements  to  evaluate  system  reliability 
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art  -failures,  operating  hours,  maintenance  events,  and  other  events  that 
render  the  system  inoperable  <5:Ch  17,  A7-4  -  A7-6) . 

Reliability  evaluation  and  estimates  during  IOT&E  may  be  difficult 
because  of  low  operating  hours.  Additionally,  point  estimates  during 
IOT&E  only  indicate  current  reliability  and  do  not  project  reliability 
of  the  mature  system.  Therefore,  the  reliability  evaluation  and 
est imates  must  include  reliability  growth  techniques  to  reliably 
represent  future  projections  <5:14i).  Because  of  limited  operating 
hours  during  IOT&E  tests  of  hardware,  evaluation  estimates  should  also 
take  into  consideration  hours  of  operation  accumulated  during 
representative  DT&E  tests  <12:91;  13:8). 

Maintainabi 1 i ty 

The  assessment  and  evaluation  of  maintainability  is  to  determine  if 
the  system  can  be  sustained  in  an  operational  state  within  a  given  level 
of  support.  Maintainability  can  be  evaluated  both  quantitatively  and 
qualitatively.  Quantitatively,  the  system  can  be  evaluated  by  looking 
at  the  average  time  a  system  is  not  capable  of  performing  its  intended 
mission  (mean  downtime  <MDT)).  Other  quantitative  measures  of 
maintainability  include  the  maintenance  manhours  required  to  repair  the 
system  and  the  average  time  to  repair  or  restore  the  system,  mean  time 
to  repair  (MTTR) .  Qualitatively,  maintainability  of  the  system  should 
be  evaluated  for  accessibility,  serviceability,  ease  of  maintenance, 
safety,  and  human  factors. 

Sources  of  data  for  maintainability  evaluation  are  repair  time  on 
failures,  number  of  personnel  performing  tasks,  questions  and  comments 
of  evaluators  and  technicians,  and  other  data  collected  for  availability 
and  reliability  evaluations.  As  with  reliability,  maintainability  may 
experience  growth  and  the  evaluation  should  include  a  growth  program  to 
project  mature  estimates  of  maintainability  <5:Ch  18,  A7-7  -  A7-9) . 

Logistics  Suopor tabi 1 i ty 

'Logistics  suppor tabi 1 i ty  assessment  addresses  the  various  elements 
of  integrated  logistics  support  (ILS).  .  .  ■  <5:48).  The  specific  areas 
usually  addressed  as  part  of  logistics  suppor tabi 1 i ty  include  support 
equipment,  technical  data,  transpor tat i on ,  packaging  and  handling 
techniques,  facilities,  and  supply  support.  The  evaluation  of  this  area 
seeks  to  determine  if  the  demands  on  the  logistics  system  are  reasonable 
and  can  be  met  by  the  proposed  support  system.  A  brief  description  of 
evaluation  methodology  for  each  of  the  above  areas  follows. 

Support  eauioment.  Individual  pieces  of  support  equipment  may  be 
given  the  same  level  of  evaluation  as  major  sub-systems  of  the  overall 
weapon  system.  Considerations  such  as  availability,  reliability,  and 
maintainability  of  the  equipment  should  be  evaluated.  Sources  for  data 
are  the  same  as  those  stated  under  methodologies  for  availability, 
reliability,  and  maintainability  for  the  major  weapon  system  <5:48-50). 
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Technical  data.  The  evaluation  of  technical  data  determines 
accuracy,  completeness,  and  usability  by  the  technicians  who  will  be 
required  to  use  it  when  the  system  is  fielded.  Much  of  the  data  for 
evaluation  should  come  from  hands-on  use  of  the  technical  data  by 
evaluators  and  technicians.  Documentation  of  errors  and  changes 
provides  evaluation  information.  The  evaluation  of  technical  data  tends 
to  be  a  very  subjective  and  qualitative  assessment  <5:53-55). 

Packaoino.  handling,  and  transportation.  Evaluation  of  packaging, 
handling  and  transportation  is  conducted  to  ensure  the  system,  sub¬ 
systems,  and  support  items  can  be  safely  and  securely  moved.  "Such 
areas  as  protection  from  weather  and  rough  handling  are  prime 
considerations*  <5:55).  When  assessing  packaging,  handling,  and 
transportation,  evaluators  should  consider  all  damaged  shipments  and 
review  applicable  procedures  for  handling  and  transportation  of 
equipment.  This  area  of  evaluation  is  also  subjective,  but  can  be 
supported  quantitatively  through  documentation  of  equipment  damaged 
during  shipping  or  handling  <5:55-57). 

Fac i 1 i t i es.  New  weapons  systems  may  require  new  or  modified 
facilities  to  support  the  system.  The  adequacy  of  these  facilities 
should  be  evaluated  to  ensure  they  meet  the  needs  of  the  maintenance  and 
operational  personnel.  Also,  if  additional  facilities  are  required  the 
deficiency  should  be  identified.  Results  of  other  measured  areas,  such 
as  reliability  and  maintainabi 1 i ty,  can  provide  evaluation  data  for  the 
facilities  evaluation.  Evaluation  of  facilities  is  more  qualitative 
than  quantitative  <5:57-58). 

Supply  support.  Spares,  new  and  repaired,  are  essential  to 
maintain  any  system  in  a  high  operational  state.  Evaluation  of  the 
supply  support  for  a  new  system  determines  how  well  the  system  is 
designed  and  planned  to  support  the  demands  of  the  weapon  system.  Data 
to  evaluate  supply  support  comes  from  reliability  data,  which  shows 
demands  on  the  system,  and  planned  provisioning  information.  High 
failure  rate  items  should  be  identified,  as  well  as  items  requiring 
additional  monitoring  or  evaluation  <5:50-53). 

Supply  support  and  all  the  other  areas  of  logistics  suppor tabi 1 i ty 
are  difficult  to  evaluate  during  IOT&E  because  of  the  early  equipment 
designs  and  lack  of  operational  equipment.  The  total  integrated 
logistics  support  system  is  seldom  complete.  Therefore,  evaluation  of 
this  area  requires  many  subjective  judgements  and  utilization  of 
projections,  contractor  estimates,  ajid  data  from  other  evaluation  areas. 


Chapter  Three 


DESCRIPTION  AND  ANALYSIS  OF 
SUITABILITY  TEST  OBJECTIVES,  SUPPORTING  TESTS, 
AND  EVALUATION  METHODOLOGIES  OF  THE  PEACEKEEPER 
IN  MINUTEMAN  SILOS  TEST  PROGRAM 


The  planned  objectives,  tests,  and  evaluation  methodologies  are 
drawn  -from  two  sources.  They  are  the  Peacekeeper  Operational  Test  and 
Evaluation  Plan  and  the  Integrated  Test  Plan  for  the  Peacekeeper  in 
Minuteman  Silos  Uleapon  System.  The  flow  of  the  test  objectives  from  the 
critical  issues  is  shown  graphically  in  Figure  2  and  described 
narratively.  The  development  of  supporting  tests  and  their  relationship 
to  the  test  objectives  is  described.  And  finally  the  evaluation 
methodologies  as  described  in  the  test  plan  are  presented  along  with 
additional  information  on  the  evaluation  methodologies. 


TEST  OBJECTIVES 


Descr i ot ion 

There  were  five  critical  operational  issues  identified  for  the 
Peacekeeper  program.  Only  two  of  those  issues  which  provide  the 
starting  point  for  the  suitability  test  object ives  wi 1 1  be  discussed. 
The  first  of  these  critical  issues  is: 


Mission  Effectiveness,  the  Peacekeeper  missile  force  must  be: 
capable  of  being  effectively  targeted  against  candidate 
targets,  available  for  operational  commitments,  and  if 
executed,  successfully  complete  assigned  mission  profiles. 
This  capability  is  best  described  by  a  factor  called  mission 
effectiveness  which  measures  the  percentage  of  deployed 
warheads  that,  if  executed,  would  produce  a  detonation  in 
intended  target  areas  discounting  the  effects  of  enemy  action. 
It  is  therefore  vital  for  the  Peacekeeper  missile  system  to 
achieve  a  sufficient  level  of  mission  effectiveness.  The 
areas  which  must  be  addressed  in  support  of  the  mission 
effectiveness  factor  are  targeting  efficiency,  alert 
availability  (SAC  operational  readiness  rate),  and  launch  and 
flight  reliability  (3:2-2). 
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The  second  critical  issue  is: 

Weapon  system  operation  and  support.  The  weapon  system  must 
be  capable  of  being  operated,  maintained,  and  logistically 
supported  throughout  its  [sic]  life  cycle.  This  issue 
encompassed  logistics  reliability,  maintainability,  support 
equipment,  transportation  and  handling,  technical  data,  supply 
support,  manpower  and  training  0:2-3). 

As  shown  in  Figure  2  -four  o-f  the  operational  suitability  objectives 
-flow  -from  these  two  critical  operational  issues. 


CRITICAL  ISSUES 


MISSION  EFFECTIVENESS 


ALERT  AVAILABILITY 


OPERATION  AND  SUPPORT 


OBJECTIVES 


•■LOGISTICS  RELIABILITY 
•MAINTAINABILITY 
-LOGISTICS  SUPPORTABI LITY 


Figure  2.  Flow  Diagram  o-f  PIMS  Test  Objectives. 


The  -first  o-f  the  initial  operational  suitability  test  objectives  is 
to  'Estimate  the  alert  availability  o-f  the  Peacekeeper  weapon  system' 
0:B-2).  This  -factor  seeks  to  determine  the  probability  a  missile  will 
be  available  when  needed.  The  mathematical  expression  given  -for 
availability  in  the  test  plan  <3:B-2)  is: 


Peacekeeper  Total  System  Hours  -  Total  Hours  O-f-f  Alert 

Alert  Availability  *  Total  System  Hours 

<100  mi ssi 1 es) 


Total  hours  o-f-f  alert  refers  to  the  total  time  the  system  is  not  capable 
of  being  launched.  These  hours  are  a  result  of  the  findings  and 
estimates  of  the  reliability,  maintainability,  and  logistics 
suppor tabi 1 i ty  objectives  <3:B-2). 

The  second  suitability  objective  is:  'Evaluate  the  logistics 
reliability  of  the  Peacekeeper  weapon  system'  <3:B-5>.  The  goal  of  this 
objective  is  '.  .  .  to  characterize  the  maintenance  and  repair 
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requirements  associated  with  the  new  Peacekeeper  missile  and  other 
equipment  changes  to  make  it  operate  in  a  Minuteman  launch  -f  ac  i  1  i  t  y  • 
<3:B-7).  Assessment  of  the  maintenance  workload  and  demands  on  the 
logistics  support  system  is  the  primary  output  of  this  objective.  These 
results  are  then  to  be  compared  with  projected  support  requirements  and 
the  differences  identified  <3;B-9>. 

The  third  suitability  objective  is  "Evaluate  the  maintainability  of 
the  Peacekeeper  weapon  system"  <3sB— 11) .  The  objective  determines  the 
"maintenance  and  repair  timelines*  to  support  the  weapon  system.  <3:B- 
11)  These  maintenance  and  repair  timelines  are  to  be  compared  with 
current  Minuteman  maintainability  factors  and  predicted  maintenance 
factors  to  look  for  differences. 

The  fourth  objective  is  logistics  suppor tabi 1 i ty  and  it  has  five 
subobjectives  < 3 : B— 1 4) .  They  are: 

1.  Evaluate  the  adequacy  of  the  support  equipment. 

2.  Evaluate  the  transportation,  packaging,  and 
handling  techniques. 

3.  Evaluate  the  adequacy  of  technical  data. 

4.  Evaluate  the  adequacy  of  facilities. 

5.  Evaluate  the  supply  support. 

This  objective  determines,  through  the  five  sub-objectives  ,  the 
supportabi 1 i ty  of  the  weapon  system  in  the  operational  environment.  For 
the  Peacekeeper  program  each  subobjective  is. evaluated  at  two  levels; 
individually  and  then  for  its  overall  impact  on  weapon  system 
availability  < 3 : B— 1 4) . 

Evaluation  of  Peacekeeper  support  equipment  focuses  on  those  pieces 
of  equipment  that  "troubleshoot,  test,  calibrate,  and  repair"  the  system 
and  other  equipment  to  maintain  the  system  (3:B-15).  Emphasis  is  placed 
on  the  qualitative  aspects  of  the  support  equipment  rather  than  the 
quantitative  measures.  This  is  shown  in  the  test  plan,  for  example, 

".  .  .  MSE  [maintenance  support  equipment]  should  be  mobile,  easily 
handled,  standardized  ..."  <3:B-15). 

Because  of  the  long  distances  between  support  bases  and  remote 
missile  launch  facilities,  evaluation  of  transportation,  packaging,  and 
handling  techniques  is  important.  The  goal  of  this  objective  is  to 
determine  if  equipment  and  systems  can  be  transported  and  handled 
safely,  without  damaging  those  systems  and  equipment.  Another 
consideration  for  this  objective  is  the  manhours  required  to  transport 
and  handle  the  equipment.  The  primary  emphasis  of  this  objective  is 
evaluation  of  procedures  for  transportation  and  handling  between  the 
support  base  and  the  launch  facility  < 3 1 B— 1 6  -  B-18). 

The  evaluation  of  technical  data  will  ".  .  .  examine  technical  data 
as  a  vital  element  of  the  Peacekeeper  weapon  system  that  affects 
virtually  every  interaction  between  operator/maintainer  and  equipment’ 
<3:B-18).  The  technical  data  will  be  judged  based  on  it's  ability  to 
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meet  *.  .  .  command  requirements  and  operational  concepts*  <3:B-20). 
Additionally,  the  technical  data  should  be  usable  and  understandable  by 
the  using  technicians.  The  test  plan  provides  a  list  o-f  other  criteria 
to  evaluate  technical  data  for  usability  by  individuals  (3:B-19). 

New  and  modified  facilities  are  being  built  to  support  the 
Peacekeeper  weapon  system.  The  subobjective  to  evaluate  the  adequacy 
of  facilities  seeks  to  determine  if  the  new  facilities  can 

.  .  .  safely  and  efficiently  (timely  completion)  support  all 

missile  processing  tasks,  missile  maintenance  activities,  and 
missile  emplacment  sequence  to  return  a  Peacekeeper  facility 
to  alert  status  following  a  missile/component  recycle 
requirement  < 3 s B— 21 > . 

The  objective  also  concerns  itself  with  the  safety  and  working 
conditions  of  the  personnel  utilizing  the  facility.  Any  deficiencies  in 
the  facilities  are  identified,  especially  those  that  impact  the 
availability  of  the  entire  weapon  system  (3:B-21). 

Evaluation  of  the  Peacekeeper  supply  support  looks  at  the  supply 
levels  and  the  failure  rates  to  see  if  differences  exist.  This 
subobjective  of  logistics  suppor tabi 1 i ty  is  closely  related  to  the  two 
major  objectives  of  logistics  reliability  and  availability.  If  a 
component  fails  (reliability)  and  spares  are  not  availability  to  replace 
the  failed  component,  the  weapons  system  will  not  be  able  to  accomplish 
its  mission  (availability).  The  test  plan  pointsout  that  the 
evaluation  of  supply  support  is  an  ongoing  and  arr  "iterative  process* 
because  of  the  'dynamic*  relationship  between  failure  rates  and  spare 
parts  levels  (3:B-22  -  B-23) . 

Analysis 


The  PIMS  test  objectives  followed  the  development  procedure 
outlined  in  Air  Force  regulations.  The  critical  operational  issues  were 
identified  and  then  supporting  test  objectives  developed.  The  test 
objectives  for  the  initial  operational  suitability  evaluation  were  the 
classic  suitability  objectives  of  availability,  reliability, 
maintainability,  and  logistics  supportabi 1 i ty.  However,  with  the 
exception  of  the  logistics  suppor tabi 1 i ty  objective,  none  of  the  other 
objectives  had  supporting  subobjectives.  Further,  specific  MOEs  were 
not  developed  for  any  of  the  four  major  suitability  test  objectives. 

The  failure  to  develop  subobjectives  and  specific  MOEs  may  have  led  to 
the  small  number  of  supporting  tests  for  operational  suitability. 


SUPPORTING  TESTS 


Descr i p  t i on 

The  PIMS  test  program  recognized  the  relationship  between  test 
objectives  and  supporting  tests,  but  also  realized  the  limitations  in 
testing  as  shown  by  the  -following  statements.  "A  major  activity  in  the 
OT&E  process  is  the  establishment  o-f  test  requirements  which  define  how 
testing  must  be  accomplished  to  meet  all  test  objectives*  <3:2-5). 
'Because  of  the  limited  and  expensive  test  assets,  the  destructive 
nature  of  ICBM  flight  tests,  and  compressed  schedule  between  major 
milestones  and  IOC  [Initial  Operational  Capability],  the  Peacekeeper 
test  and  evaluation  effort  is  a  combined  DT&E/OT&E  program*  (3:21). 
However,  the  operational  test  planners  made  a  decision  to  place  emphasis 
on  tests  supporting  the  operational  effectiveness  test  objectives  over 
suitability  objectives  <3:2-6). 

Tests  to  support  the  10T&E  suitability  test  objectives  are  found  in 
the  "Weapon  System  Testing*  section  of  the  Integrated  Test  Plan  <ITP). 
‘Peacekeeper  weapon  system  tests  are  conducted  on  operationally 
representative  equipment  and  facilities  to  demonstrate  system  compliance 
with  operational  requirements*  <6:1-6).  Within  the  weapon  system  tests, 
44  tests  support  the  four  suitability  test  objectives  of  availability, 
reliability,  maintainability,  and  logistics  suppor tabi 1 i ty .  Many  of  the 
tests  more  than  one  of  the  objective.  There  are  11  tests  tied  to 
availability,  one  test  tied  to  logistics  reliability,  41  tests  tied  to 
maintainability,  and  32  tests  tied  to  logistics  suppor tabi 1 i ty .  The 
flight  tests  support  operational  suitability  objectives,  but  they  are 
not  shown  as  supporting  the  objectives  in  the  ITP.  The  majority  of  the 
tests  tied  to  the  suitability  objectives  are  tests  to  evaluate  the 
performance  of  maintenance  tasks  and  the  maintainability  objective. 

The  ITP  also  plans  for  the  involvement  of  the  IOTicE  test  team  in 
the  testing.  "The  primary  purpose  for  AFOTEC  participation  is  to 
satisfy  OT&E  objectives  and  requirements*  <6:1-51).  Test  team 
involvement  will  gradually  increase  from  “observation  to  involvement  to 
performance"  <6:1-53).  The  performance  of  the  maintenance  and 
operations  tasks  shifting  from  contractors  to  Air  Force  personnel 
provides  a  more  realistic  operational  environment  for  testing  <6:1-51  - 
1-53) . 

Analysis 

The  emphasis  on  the  tests  supporting  operational  suitability 
objectives  is  directed  at  the  maintainability  objective.  Forty-one  of 
the  44  tests  tied  to  the  suitability  objectives  are  tests  in  which 
maintenance  tasks  are  performed.  The  few  remaining  tests  support 
technical  data  or  measure  the  operational  suitability  for  operations 
personnel  versus  maintenance  tasks. 

In  reviewing  the  tests  and  the  objectives  they  are  tied  to,  some 
relationships  are  questionable.  It  appears  that  rather  than  the  tests 
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being  developed  to  support  test  objectives,  tests  were  developed  and 
then  tied  to  an  objective  it  might  support.  An  example  is  the  logistics 
reliability  objective  and  the  single  test  tied  to  the  objective.  The 
test  is  an  evaluation  of  the  Expanded  Maintenance  Data  Acquisition 
System  <EMDAS),  a  data  collection  system  already  in  place  at  Minuteman 
bases.  The  system  is  used  to  collect  -fault  i  n -forma  t  i  on  and  maintenance 
responses  and  will  eventually  be  used  by  the  Peacekeeper  weapon  system. 
The  EMDAS  system  has  no  e-f-fect  on  the  reliability  o-f  the  Peacekeeper 
weapon  system.  It  is  also  unusual  that  the  -flight  tests  are  not  shown 
as  supporting  the  logistics  reliability  objective,  when  in  -fact  these 
tests  provided  the  primary  source  o-f  operationally  relevant  -failure  data 
•for  the  logistics  reliability  assessment. 

The  test  planners  did  have  the  -foresight  to  call  -for  the 
involvement  o-f  Air  Force  personnel  to  per-form  tasks  in  the  operational 
environment,  utilizing  operational  hardware  and  procedures.  The  ITP 
shows  the  plan  -for  this  involvement  through  a  very  speci-fic 
participation  matrix  detailing  how,  when,  and  to  what  degree  Air  Force 
test  personnel  would  participate  <6:1-54  -  1-56).  One  test  also  shows 
the  attempt  to  expose  the  weapon  system  and  associated  hardware  to  the 
varied  operational  environments.  This  adverse  weather  test  measures  the 
per-formance  o-f  maintenance  in  cold  weather,  and  satis-fies  the  need  -for 
making  the  tests  as  operationally  relevant  as  possible. 

The  test  plan  <3:1-10)  also  outlines  the  gradual  shi-ft  in  test 
emphasis  -from  DT&E  to  OT&E.  AFR  80-14  and  DOD  Directive  5000.3  state,  . 
"Any  combined  test  program  chosen  shall  contain  enough  dedicated 
operational  test  events  to  satis-fy  the  operational  testing  agency 
requirements  -for  an  independent  evaluation*  (9:24;  10:8).  The  -final 
phase  o-f  testing  and  test  -flights  were  planned  -for  operational  testing 
and  the  emphasis  placed  on  0T&E  tests.  However  to  date,  the  final  three 
test  flights  have  not  taken  place. 

In  conclusion,  tests  to  support  all  operational  suitability 
objectives  are  not  present  in  the  plan.  The  tests  that  were  included, 
however,  were  well  designed  in  that  they  emphasized  operationally 
realistic  environments,  used  operational  hardware  and  procedures,  and 
were  performed  by  the  people  required  to  operate  them  the  system  when 
deployed. 


EVALUATION  METHODOLOGIES 

The  evaluation  methodologies  utilized  by  the  PIMS  test  program  were 
not  necessarily  in  line  with  those  recommended  by  the  current  guidance. 
One  reason  for  this  may  have  been  the  lack  of  supporting  tests  and  MOEs 
tying  tests  to  objectives.  The  avai 1 abi 1 i ty  model  in  the  test  plan 
provides  one  of  the  more  specific  methodologies  presented. 


Estimation  of  Peacekeeper  alert  availability  requires  examination 
of  the  three  factors  affecting  it.  The  first  factor  is  the  reason  the 
system  will  be  unable  to  launch  and  fulfill  its  intended  mission. 
Component  failure  is  a  primary  factor  affecting  alert  availability. 


Other  reasons  preventing  launch  commitment  are  periodic  maintenance, 
tests,  and  owning  command  requirements.  The  second  factor  affecting 
availability  is  how  long  it  takes  to  restore  the  system  when  it  is  not 
launch  capable.  This  factor  includes  actual  repair  times,  travel  times, 
and  any  other  factors  that  may  delay  restoring  the  weapon  system.  The 
third  factor  is  support  items  influencing  availability.  Areas  examined 
in  this  factor  include  " .  .  .  manpower,  tools,  vehicles,  and  spares  .  . 

.*  <3:B-4).  The  source  of  data  for  the  assessment  of  availability  comes 
from  contractor  predictions,  current  Minuteman  maintenance  and 
reliability  data,  and  the  results  of  the  other  three  suitability 
objectives  of  reliability,  maintainability,  and  logistics 
supportabi 1 i ty .  The  results  of  the  objectives  and  other  assessments  are 
placed  into  a  model,  which  is  shown  in  Figure  3.  The  final  output  of 
the  availability  objective  is  an  estimate  of  alert  availability.  This 
estimate  is  then  compared  to  the  using  command's  required  availability 
<3:B-3  -  B-5). 

Analysis  of  Availability 

The  guidance  for  evaluation  of  availability  clearly  recognizes  the 
difficulty  in  evaluating  availability  and  recommends  the  use  of  modeling 
or  simulation  <5:Ch  16,  A7-1  -  A7-2) .  While  the  PIMS  planners 
recognized  the  limitations  in  collecting  data  to  support  the 
availability  analysis,  no  simulation  or  modeling  was  planned  nor  was  it 
ever  used.  Also,  the  planners  did  not  use  standard,  acceptable  terms  or 
methods  in  the  availability  model.  For  example,  they  chose  to  use 
"number  of  annual  occurrences"  rather  than  mean  time  between 
replacement  (MTBR)  or  mean  time  between  maintenance  <MTBM)  and  "time  to 
restore"  instead  of  using  the  standard  term  of  mean  downtime  (MDT) .  The 
estimate  produced  by  the  model  is  the  same,  but  it  is  not  a  straight 
forward  method. 

Description  of  Looistics  Reliability 

Evaluation  of  logistics  reliability  involves  examining  what  each 
contractor  says  his  system  will  do  and  measuring  this  against  what 
it  actually  does  during  testing.  The  evaluators  look  for  discrepancies 
between  what  is  predicted  and  what  is  observed.  These  differences  are 
reported  and  used  to  revise  predictions  that  characterize  system 
failure.  Emphasis  is  placed  on  using  the  Logistics  Support  Analysis 
Record  (LSAR)  as  the  source  and  reporting  document.  Actual  failure  data 
is  collected  by  the  test  team  and  used  for  analysis.  Other  outputs  of 
this  objective  are  used  to  determine  high  failure  rate  items,  safety 
concerns,  and  high  manhour  consuming  items.  These  are  then  converted 
into  yearly  occurrences  for  inclusion  in  the  availability  model  (Figure 
3)  <3:8-5  -  B-l 1 ) . 
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Figure  3.  An  Example  0-f  Predictive  Off-Alert  Model  <3:B-6) 


The  evaluation  of  the  weapon  system's  logistics  reliability  is 
again  not  necessarily  within  the  guidance  provided.  There  were  no 
specific  MOEs  and  no  specific  tests  to  support  this  objective. 

Evaluation  relied  largely  on  contractor  provided  evaluations  and  data. 
Related  to  this  dependence  on  contractor  data  was  the  fact  that  the  PIMS 
program  did  not  have  a  centralized  data  collection  system  as  required  by 
the  guidance.  A  limited  data  collection  system  was  developed  by  the 
test  team  but  did  not  include  inputs  from  any  other  sources.  Much  of 
the  data  used  to  evaluate  the  logistics  reliability  of  the  system  came 
from  operation  of  the  system  prior  to  a  flight  test.  These  periods  were 
relatively  short  (an  average  of  seven  days)  when  compared  to  the 
continuous  run  requirements  of  the  operational  system.  Also,  no  use  of 
reliability  growth  techn i ques  was  incorporated  in  the  test  plan.  The 
use  of  reliability  growth  techniques  is  recommended  in  the  current 
guidance  (5:141)  and  would  have  been  appropriate  because  of  the  high 
operating  time  requirements  and  the  low  number  of  actual  hours  during 
test i ng. 

Description  of  Maintainability 

As  with  reliability,  maintainability  is  assessed  by  examining  what 
each  contractor  estimates  the  maintenance  requirements  are,  compared  to 
what  is  observed  by  the  test  team.  This  objective  is  supported  by  41 
tests  of  which  the  majority  are  the  actual  performance  of  maintenance 
tasks  by  test  team  personnel.  Differences  between  the  contractor 
predictions  and  actual  performance  are  reported.  The  sources  of  data 
are  contractor  reports,  the  LSAR,  and  actual  performance  data  collected 
by  the  test  team.  The  evaluation  looks  for  high  manpower  consuming 
tasks,  tasks  taking  a  long  time  to  perform,  and  methods  to  improve  and 
shorten  tasks.  Also,  timelines  are  developed  to  input  into  the 
avai l abi 1 i ty  model  as  time  to  restore  the  system.  These  timelines 
include  all  applicable  travel  times,  delays,  and  maintenance  actions 
(3:8-11  -  B-l 4) . 

Analysis  of  Maintainability 

The  maintainability  objective  is  well  supported  by  actual  tests. 

The  evaluation  provides  data  to  make  a  quantitative  assessment  of  the 
weapon  system's  maintainability  and  also  allows  a  qualitative  evaluation 
by  Air  Force  personnel  performing  the  tests.  However,  no  specific  MOEs 
were  established  for  this  objective  to  link  the  tests  and  the  major 
objective.  Standard  maintainability  terms,  such  as  MDT  and  MTTR  were 
not  used  in  the  description  of  the  methodology.  As  with  logistics 
reliability,  no  maintainability  growth  techniques  were  planned  for  or 
used  by  the  test  team. 

Description  of  Logistics  Suoportabi 1 i ty 


Each  of  the  five  subobjectives  of  the  logistics  suppor tabi 1 i ty 
objective  has  a  separate  methodology  in  the  test  plan.  The  first  sub- 
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objective  of  support  equipment  takes  a  limited  approach  to  the 
evaluation.  No  specific  tests  were  developed  to  support  this 
evaluation,  but  rather  evaluation  takes  place  only  when  the  equipment  is 
actually  needed  due  to  a  failure  or  planned  task.  Much  of  the  analysis 
will  be  accomplished  by  design  reviews  and  "table-top*  reviews  of 
support  equipment  procedures.  Sources  for  data  come  from  contractors' 
designs,  performance  data,  and  test  team  observations.  Support 
equipment  affecting  availability  is  highlighted  for  further  study  and 
analysis,  and  subjective,  qualitative  assessments  of  support  equipment 
are  also  made  <3:B-14  -  8-16). 

The  methodology  to  evaluate  transportation,  packaging,  and  handling 
is  supported  by  specific  tests  of  the  operational  procedures  used  to 
handle  and  transport  major  components  of  the  weapon  system.  Test  team 
members  perform  transportation  and  handling  tasks,  report  any  problems, 
and  record  actual  times  to  perform  the  tasks  for  comparison  with 
predicted  contractor  times.  Additionally,  the  evaluation  makes  a 
qualitative  assessment  of  the  transportation,  packaging,  and  handling 
procedures  <3:B-16  -  B-18). 

The  method  used  to  evaluate  technical  data  is  taken  from  operators 
and  technicians  using  the  technical  data  to  prepare  missiles  for  the 
flight  test  program.  If  operational  technical  data  is  not  yet  available 
for  use,  the  test  team  will  participate  in  technical  order  reviews, 
validations,  and  verifications  <3:B-18  -  B-20).  The  evaluation  compares 
the  technical  data  to  the  "...  using  command  requirements  and 
operational  concept"  <3:B-20). 

While  no  specific  test  is  dedicated  to  the  facilities  sub¬ 
objective,  many  of  the  maintenance  tasks  must  be  performed  in  the  new  or 
modified  facilities  at  both  the  operational  base  and  the  test  base.  The 
test  team  also  participates  in  facility  design  reviews  to  assess  their 
adequacy  in  meeting  operational  needs  <3:B-20  -  B-21). 

To  evaluate  supply  support,  actual  failure  rates  of  the  components 
are  compared  to  the  projected  supply  levels.  If,  based  on  the  failure 
rates,  a  shortfall  exists  in  the  supply  levels,  it  is  reported. 
Shortfalls  in  the  supply  levels  can  impact  the  availability  of  the 
weapon  system,  so  any  shortfall  may  require  further  study.  Because  of 
this,  the  test  plan  calls  for  an  ongoing  evaluation  of  supply  support 
<3:B-22  -  B-23) . 

Analysis  of  Logistics  Supoor tabi 1 i ty 

The  logistics  suppor tabi 1 i ty  objective  had  supporting  tests  which 
provided  data  for  evaluation.  Also,  the  numerous  maintainability  test 
events,  requiring  use  of  the  maintenance  support  equipment,  supported 
the  evaluation  of  the  support  equipment.  The  test  plan  methodology  does 
not  establish  any  requirements  to  measure  availability,  reliability,  or 
maintainability  of  the  maintenance  support  equipment;  however,  since  it 
impacts  availability  it  was  done  by  the  evaluators  conducting  the 
evaluation  of  the  support  equipment.  Technical  data  was  supported  in 
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much  the  same  way  as  support  equipment.  Air  Force  technicians 
performing  the  maintenance  and  operations  tasks  used  the  operational 
technical  data.  They  were  able  to  make  a  qualitative  assessment  o-f  the 
technical  data  and  recommend  changes.  The  areas  evaluated  were  in  line 
with  those  outlined  in  the  current  guidance.  The  packaging,  handling, 
and  transportation  of  the  Peacekeeper  weapon  system  was  supported  by 
tests  of  the  system.  The  tests  measured  how  well  the  procedures 
protected  the  system  from  damage  and  the  elements.  The  criteria  used  to 
evaluate  the  procedures  were  in  accordance  with  the  current  guidance. 
Again,  as  with  support  equipment  and  technical  data,  the  numerous 
maintenance  tests  supported  the  evaluation  of  the  new  and  modified 
Peacekeeper  facilities.  The  methodology  to  evaluate  supply  support  is 
in  accordance  with  the  guidance;  however,  because  of  its  close 
relationship  with  the  major  objective  of  logistics  reliability,  the 
evaluation  of  this  sub-objective  is  affected.  If  the  logistics 
reliability  objectives  does  not  provide  adequate,  accurate  data,  the 
supply  support  objective  will  probably  not  be  adequate  or  accurate. 


CONCLUSION 

The  overall  methodology  to  evaluate  operational  suitability  is 
characterized  by  a  dependence  on  maintainability  tests  and  too  few  tests 
to  support  the  logistics  reliability  objective.  This  results  in  over 
dependence  on  subjective  evaluations.  The  lack  of  reliability  and 
maintainability  growth  techniques  also  detracts,  and  is  in  conflict  with 
current  guidance.  'Testing  shall  be  planned  and  conducted  to  provide 
quantitative  data  and  to  minimize  the  need  for  subjective 
interpretations  of  system  performance*  <9:2-3) . 


Chapter  Four 


CONCLUSION  AND  RECOttiENDATI ONS 


CONCLUSION 


Shortcomings  in  the  current  guidance  prevent  optimum  conduct  of 
initial  operational  suitability  test  and  evaluation  of  land  based  ICBMs. 
As  pointed  out  in  the  recommendations,  guidance  specifically  designed  to 
develop  suitability  MOEs  is  needed  to  provide  the  link  between 
objectives  and  tests.  The  most  serious  shortcoming  is  the  lack  o-f 
emphasis  and  guidance  to  develop  tests  to  support  suitability 
objectives.  Uith  today's  increased  emphasis  on  reliability  and 
maintainability,  realistic  suitability  testing  is  essential  -for  all 
weapon  systems.  Simulation  and  modeling  are  helpful,  but  require  too 
many  assumptions  and  subjective  evaluations.  Evaluation  methodologies 
designed  for  land  based  ICBMs  are  needed.  Current  specific  guidance 
available  in  AFOTECP  400-1,  is  inadequate  as  it  does  not  provide 
guidance  for  ICBMs.  An  ICBM  annex  is  required  for  this  directive. 


When  compared  with  the  guidance  available,  the  PIMS  test  program 
was  hampered  by  the  lack  of  guidance.  In  some  cases,  the  test  planners 
did  not  follow  the  guidance  available.  Whether  this  was  deliberate  or 
not  cannot  be  determined  by  this  author  and  is  outside  the  limits  of 
this  study.  The  failure  to  utilize  any  kind  of  simulation  or  modeling 
or  reliability  growth  techniques1  are  two  of  the  more  notable 
exceptions  to  the  guidance.  There  was  no  centralized  data  collection 
system  in  the  PIMS  program  and  the  guidance  is  very  specific  in  the  need 
for  a  centralized  data  collection  system.  This  made  evaluation  of 
logistics  reliability  and  maintainability  difficult  for  the  operational 
test  team.  Other  factors  such  as  the  compressed  schedule,  cost, 
indecision  on  basing,  and  Presidential  and  Congressional  interest  drove 
the  test  program. 


Future  ICBM  IOT&E  testing  will  become  even  more  complicated  then 
the  Peacekeeper  program  because  of  the  requirement  for  new  mobile  launch 
platforms.  Peacekeeper  Rail  Garrison  and  the  Small  ICBM  are  both  mobile 
systems.  The  suitability  testing  for  these  systems  presents  the  test 
planner  with  new  challenges  for  developing  test  objectives,  supporting 
tests,  and  evaluation  methodologies  to  ensure  the  Air  Force  receives 
reliable  and  maintainable  weapon  systems. 

1  Reliability  growth  technique  was  employed  for  the  Missile 
Guidance  and  Control  Set  <MGCS) . 


RECOMMENDATIONS 

Suitability  Test  Objectives 

The  analytical  process  to  develop  suitability  objectives  'from  the 
critical  operational  issues  provides  an  adequate  means  to  develop  test 
objectives,  as  the  major  suitability  objectives  are  standard  -for 
virtually  any  weapon  system.  The  area  requiring  additional  guidance  -for 
1CBM  test  objectives  is  MOEs.  ICBM  missions  are  not  varied;  their 
challenge  is  in  maintaining  a  constant  state  o-f  readiness.  The  MOE 
process  outlined  in  the  guidance  is  more  oriented  to  aircra-ft  type 
'mission  e-f-fect i veness'  rather  than  ICBM  slanted  "suitability." 

Overall,  the  guidance  to  develop  test  objectives  is  su-f-ficient  -for  any 
weapon  system,  including  ICBMs,  but  lacking  in  the  development  o-f  ICBM 
MOEs  to  tie  objectives  to  tests. 

Support i no  Tests 

The  emphasis  in  the  current  guidance  is  on  the  development  o-f 
mission  effectiveness  tests  and  evaluations.  AFR  55-43  provides  a 
process  to  develop  tests  supporting  mission  effectiveness  but  the 
corresponding  section  under  suitability  test  development  does  not 
provide  any  process.  Its  only  recommendation  is  to  use  simulation  and 
modeling.  This  omission  appears  to  affect  not  only  tests  to  support 
ICBM  suitability  testing  but  other  systems  as  well.  Guidance  to  develop 
tests  to  support  suitability  test  objectives  should  be  expanded  to  be  at 
least  as  detailed  as  those  for  mission  effectiveness.  Also,  in 
considering  tests  to  support  ICBM  suitability  testing,  the  guidance 
should  consider  the  operation  of  ICBMs  and  the  requirements  for  extended 
system  running  times  and  maintenance  demands. 

Evaluation  Methodologies 

AFOTECP  400-1  Looistics  Assessment,  provides  detailed  descriptions 
of  each  of  the  suitability  objectives  and  methodologies  to  evaluate  the 
objectives  of  availability,  reliability,  maintainability  and  logistics 
supportabi 1 i ty .  However,  the  only  reference  to  missiles  is  in  an 
appendix  for  "munitions  and  missiles"  referring  to  missiles  used  with 
aircraft.  There  is  no  corresponding  appendix  for  land-based  ICBMs.  An 
appendix  for  land  based  ICBMs  is  needed  to  support  suitability 
evaluation  methods  of  these  systems. 

Other 


The  current  process  of  test  and  evaluation  is  very  dependent  on  the 
requirements  established  by  the  Air  Force  agency  requesting  the  new 
system.  While  researching  the  current  guidance,  a  new  Air  Force 
Pamphlet  was  found  which  should  have  an  impact  on  testing  to  determine 
operational  suitability.  AFP  57-9,  Defining  Looistics  Requirements  in 


The  best  opportunity  tor  influencing  design,  -from  a  readiness 
and  logistics  perspective,  is  at  the  front  end  (preconceptual 
and  conceptual  phases)  of  the  acquisition  process.  The  most 
effective  is  by  establishing  specific,  realistic  qualitative 
and  quantitative  readiness  and  logistics  requirements  in  the 
SON  <10:1). 

This  pamphlet  also  has  specific  guidance  for  strategic  land  based  ICBMs 
logistics  requirements  such  as  a  measure  for  maintenance  manhours  per 
operational  unit,  mean  down  time,  mean  time  to  repair,  and  mean  time 
between  maintenance  <10:18).  The  recognition  that  ICBMs  require 
separate  consideration  and  should  not  be  considered  with  munitions  or 
aircraft  is  a  step  in  the  right  direction. 
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APPENDIX 


APPENDIX  A 
Def i n i t i ons 

Definitions  of  key  terms  are  presented  so  the  reader  will  have  an 
understanding  of  the  terminology  used  in  this  paper.  Definitions 
presented  here  are  standard  definitions  used  throughout  the  Air  Force 
and  Department  of  Defense  and  are  taken  from  representative  regulations. 
Where  definitions  are  broader  than  the  scope  of  this  paper,  those 
definitions  are  so  annotated. 

Avai 1 abi 1 i ty.  A  measure  of  the  degree  to  which  an  item  is  in  the 
operable  and  committable  state  when  the  mission  is  called  for  at  any 
random  point  in  time.  Availability  is  dependent  on  reliability, 
maintainability,  and  logistics  supportabi 1 i ty. 

<ICBM  Availability)  The  percent  of  a  missile  force  capable  of 
commitment  to  the  launch  sequence  at  any  random  point  in  time.  (5:A2-3) 

Critical  Issues.  Those  aspects  of  a  system's  capability,  either 
operational,  technical,  or  other,  that  must  be  questioned  before  a 
system's  overall  worth  can  be  estimated,  and  that  are  of  primary 
importance  to  the  decision  authority  in  reaching  a  decision  to  allow  the 
system  to  advance  into  the  next  acquisition  phase.  (9:34) 

Initial  Operational  Test  and  Evaluation  (IOT&E).  The  first  phase  of 
operational  test  and  evaluation  conducted  on  preproduction,  prototypes,- 
or  pilot  production  items  and  normally  completed  prior  to  the  first 
major  production  decision.  It  is  conducted  to  provide  a  valid  estimate 
of  a  system's  operational  effectiveness  and  operational  suitability 
prior  to  the  first  major  production  decision.  (9:35) 

Logistics  Supportab i 1 i ty ■  How  well  the  composite  of  support 
considerations  necessary  to  achieve  the  effective  and  economical  support 
of  a  system  or  equipment  for  its  life  cycle  meets  stated  quantitative 
and  qualitative  requirements.  This  includes  integrated  logistics 
support  and  logistics  related  O&S  cost  considerations.  <5:A2-11) 

Maintainabi 1 i ty.  The  measure  of  the  ability  of  an  item  to  be  retained 
in  or  restored  to  specified  condition  when  maintenance  is  performed  by 
personnel  having  specified  skill  levels,  using  prescribed  procedures  and 
resources,  at  each  prescribed  level  of  maintenance  and  repair.  <5:A2-11) 

Operational  Suitability.  The  degree  to  which  a  system  can  be 
satisfactorily  place  in  field  use  with  consideration  being  given  to 
availability,  compatibility,  transportabi 1 i i ty,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportabi 1 i ty,  logistic  suppor tabi 1 i ty ,  and  training 
requirements.  (9:34) 

NOTE:  For  consideration  in  this  paper  operational  suitability  is 
limited  to  availability,  logistics  reliability,  maintainability,  and 
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logistics  suppor tab i 1 i ty .  Those  areas  that  do  not  overlap  into 
ettec t  i  veness .  (5:5) 

Operational  Test  and  Evaluation  (QT&E).  OT&E  is  conducted  in  as 
realistic  conditions  as  possible  throughout  the  system  lite  cycle.  It  is 
done  to  estimate  (or  to  re-fine  estimates  o-f)  a  system's  operational 
e-f-fect  i  veness  and  operational  suitability,  to  identity  any  operational 
det i c i enc i es ,  and  to  identity  the  need  tor  any  mod i t i cat i ons .  <5:A2-15) 

Reliability.  The  probability  that  an  item  will  pertorm  its  intended 
tunction  tor  a  specitied  interval  under  stated  conditions. 

(Log istics  Reliability)  A  measure  ot  a  system's  capability  to  operate  as 
planned  under  the  detined  operational  and  support  concepts  using 
specitied  logistics  resources  (tor  example,  spares  or  manpower). 
Logistics  reliability  recognizes  the  ettect  ot  all  occurrences  that 
place  a  demand  on  the  logistics  support  system  even  when  mission 
capability  is  unattected.  ( 5 : A2— 17) 

Test  and  Evaluation.  The  term  “test"  denotes  any  project  or  program 
designed  to  obtain,  verity,  and  provide  data  tor  the  evaluation  ot 
research  and  development  other  than  laboratory  experiments;  Progress  in 
accomplishing  development  objectives;  pertormance  and  operational 
capability  ot  system,  subsystems,  components;  and  equipment  items.  The 
term  "evaluation"  denotes  the  review  and  analysis  ot  quantitative  data 
produced  during  current  or  previous  testing,  data  obtained  trom  test 
conducted  by  other  government  agencies  and  contractors,  torm  operation 
and  commercial  experience,  or  combinations  thereot.  (5:A2-19) 


AFOTEC 

AFOTECP 

AFP 

AFR 


EM  DAS 


FOT&E 


I  CBM 

IOT&E 

ITP 


HDT 

HOE 

HSE 

HTBF 

HTBCF 

HTTR 

HTBM 
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APPENDIX  B 

Abbreviations  and  Acronyms 
A 

Air  Force  Operational  Test  and  Evaluation  Center 
AFOTEC  Pamphlet 
Air  Force  Pamphlet 
Air  Force  Regulation 

D 

Department  of  Defense 

\ 

E 

Expanded  Minuteman  data  analysis  system 

F 

Follow-on  operational  test  and  evaluation 

I 

Intercontinental  ballistic  missile 
Initial  operational  test  and  evaluation 
Integrated  test  plan 

L 

Logistics  support  analysis  record 

M 

Mean  downtime 

Measure  of  effectiveness 

Maintenance  support  equipment 

Mean  time  between  failure 

Mean  time  between  critical  failure 

Mean  time  between  replacement 

Mean  time  between  maintenance 

0 

Operational  test  and  evaluation 

P 

Peacekeeper  in  Minuteman  silos 


m 

9. 

CONTINUED 


SON 


S 

Statement  o-f  Operational  Need 
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